HDDS-15864. Branch merge checklist for Short Circuit read by ChenSammi · Pull Request #503 · apache/ozone-site

ChenSammi · 2026-07-22T14:40:01Z

What changes were proposed in this pull request?

Add branch merge checklist for short-circuit read.

What is the link to the Apache Jira?

HDDS-15864

How was this patch tested?

Gargi-jais11

Thanks @ChenSammi for preparing this. LGTM!

priyeshkaratha · 2026-07-23T17:10:34Z

+
+## 2. Documentation
+
+[User Documentation](https://github.com/apache/ozone/blob/HDDS-10685/hadoop-hdds/docs/content/feature/Short-Circuit-Read.md) of Short Circuit Read has been added.


@ChenSammi @Gargi-jais11
This PR completely removes the old website and Hugo since it is now unused. So I suggest to use https://ozone.apache.org/docs/next/administrator-guide/configuration/performance/short-circuit-local-reads

priyeshkaratha

Thanks @ChenSammi for the working on the checklist. Please check the inline comment. Also we may need to create followup Jira to fix urls that pointing to feature branch code to master branch after merge.

ChenSammi · 2026-07-24T02:35:45Z

Thanks @ChenSammi for the working on the checklist. Please check the inline comment. Also we may need to create followup Jira to fix urls that pointing to feature branch code to master branch after merge.

I plan to merge this patch after feature branch is merged, so URL can be updated later.

priyeshkaratha

Thanks @ChenSammi for clarifying comments. Changes LGTM

errose28

Thanks for adding this. Left some comments inline.

errose28 · 2026-07-24T15:30:49Z

+
+## 9. Possible incompatible changes/used feature flag
+
+Short-circuit read is gated by the non-rolling upgrade framework as HDDSLayoutFeature.SHORT_CIRCUIT_READS (layout version 11). 


Why is the layout feature necessary? It looks like this just creates a socket and closes it on shutdown. How would this break on downgrade?

Also it looks like new clients can request short circuit read from an old server it will be ignored. This seems fine since the field is phrased as "request" but that is worth noting in this section too.

It's to avoid a new client talk to an old server. Yes, the new client has fallback mechanism, but it will cost time.

Then this is not correct and was not tested. It is always the newest component's responsibility to handle compatibility, and Datanode layout feature is never communicated to the client so it is impossible for a new client to do anything about this with the current implementation.

If we actually want the new client to do some handling with an old server, the layout feature should be removed and a DatanodeVersion needs to be added instead with a check on the client side. However, SCR may not even be enabled on the server side regardless of its version and finalization status, so we would need a transparent fallback anyways. The DatanodeVersion would just save a retry as a regular read for the new client case. It looks like a LayoutFeature is not required either way since I don't see any downgrade implications.

The two version system is confusing, that's why we've reduced it to a single version in ZDU so this type of thing should be easier to follow going forward.

Datanode layout feature is never communicated to the client so it is impossible for a new client to do anything about this with the current implementation

Client knows the Datanode version and checks it.

XceiverClientManager L211.

if (OzoneNetUtils.isAddressLocal(addr) && dn.getCurrentVersion() >= SHORT_CIRCUIT_READS.toProtoValue()) {

Ok this is good but the doc doesn't mention a new DatanodeVersion was also added. Looks like action items are:

Update the doc to mention the new DatanodeVersion to handle the new client case.

Remove the added HDDSLayoutFeature from the branch since there doesn't appear to be any downgrade concerns.

errose28 · 2026-07-24T15:37:52Z

+
+End-to-end read performance is therefore dominated by local disk I/O, similar to a direct file read, rather than by gRPC data transfer.
+
+A benchmark was run against feature branch HDDS-10685, and get this result https://docs.google.com/document/d/1xmjCbK4rP287pPN0_iOonnQee6HgwSI3h5hVbtgfu3k/edit?tab=t.0. 


This doc isn't very large, we can put its content directly in this file. We should not commit links to google docs which can disappear or lose access to git.

Can you convert the images to markdown tables so they show up in text searches and can have their content copied? If google doc's "copy as markdown" doesn't support this then AI should make this easy.

I already tried convert the image to markdown table first with Cursor. But cursor did not give me a nice table as I expected after several round of interaction, so finally it's converted to png image.

@errose28 , do you know how we can make this new engine allows words like YCSB and ycloud?

docs/08-developer-guide/04-project/01-git/03-feature-branches/03-merged-branches/17-hdds-10685-short-circuit-read.md:80:122 - Unknown word (YCSB) -- ozone fs` command and YCSB on a ycloud cluster docs/08-developer-guide/04-project/01-git/03-feature-branches/03-merged-branches/17-hdds-10685-short-circuit-read.md:80:132 - Unknown word (ycloud) -- command and YCSB on a ycloud cluster. Short-circuit

@ChenSammi u can add the unknown words coming up in this file cspell.yaml. Then the error will be gone.

+1 to the above, or inline cspell ignores can be added if the word is only relevant for this page. In this case YCSB is probably valid to allow globally in cspell.yaml.

The instructions in the contributing guide are also linked from the CI failure itself:

... CSpell: Files checked: 1347, Issues found: 5 in 1 file. Checking file names... CSpell: Files checked: 1, Issues found: 0 in 0 files. Spell check failed. See CONTRIBUTING.md for help fixing false positive spelling errors: https://github.com/apache/ozone-site/blob/master/CONTRIBUTING.md#spelling

Google docs "copy as markdown" was able to create the markdown tables. They render well in docusaurus too. I've put this and the spelling fix on my fork here. You can cherry-pick these into this PR.

Note I added "YCSB" to cspell where previously it was wrapped in code blocks. Although spelling is ignored for code blocks YCSB is not used as code here, and code blocks as headings render badly in docusaurus right now.

errose28 · 2026-07-24T15:44:16Z

+
+Short-circuit communication uses a UNIX domain socket (`ozone.domain.socket.path`) between the client and Datanode. 
+
+It follows the same [Hadoop Socket Path Security](https://cwiki.apache.org/confluence/spaces/HADOOP2/pages/120730260/SocketPathSecurity) rules as HDFS short-circuit reads.


Do we have a copy of this in our own docs (not a link to the hadoop docs)? Hadoop 2 wiki is falling apart and we should not be directing users there.

agreed, as a TODO item we should copy this to our doc site.

https://cwiki.apache.org/confluence/spaces/HADOOP2/pages/120730260/SocketPathSecurity content is included in the design document "Security" section.

Let's update this section to reference our own user or design docs with this content instead of linking to the hadoop wiki.

jojochuang · 2026-07-24T17:38:57Z

+
+## 2. Documentation
+
+[User Documentation](https://github.com/apache/ozone/blob/HDDS-10685/hadoop-hdds/docs/content/feature/Short-Circuit-Read.md) of Short Circuit Read has been added.


the feature would require topology awareness to maximize locality https://ozone.apache.org/docs/next/administrator-guide/configuration/performance/topology/#optimizing-read-paths

let's update user doc post merge.

jojochuang · 2026-07-28T07:23:05Z

+
+End-to-end read performance is therefore dominated by local disk I/O, similar to a direct file read, rather than by gRPC data transfer.
+
+A benchmark was run against feature branch HDDS-10685, testing short-circuit performance with the `ozone fs` command and YCSB on a ycloud cluster. Short-circuit read was toggled with `ozone.client.read.short-circuit` and `ozone.domain.socket.path`.


please replace ycloud with "Cloudera's internal test bed"

Move benchmark images under static/img, escape YCSB for spell check, and replace internal cluster name with plain-language wording. Co-authored-by: Cursor <cursoragent@cursor.com>

ChenSammi force-pushed the master branch from 21445ef to d2773c9 Compare July 22, 2026 15:21

ChenSammi marked this pull request as ready for review July 22, 2026 16:06

HDDS-15864. Branch merge checklist for Short Circuit read

f7e5fe6

ChenSammi force-pushed the master branch from d2773c9 to f7e5fe6 Compare July 23, 2026 08:17

Gargi-jais11 approved these changes Jul 23, 2026

View reviewed changes

priyeshkaratha reviewed Jul 23, 2026

View reviewed changes

priyeshkaratha approved these changes Jul 24, 2026

View reviewed changes

errose28 reviewed Jul 24, 2026

View reviewed changes

jojochuang approved these changes Jul 24, 2026

View reviewed changes

address comments

1b302c6

jojochuang reviewed Jul 28, 2026

View reviewed changes

HDDS-15864. Fix CI for short-circuit merge checklist doc.

7235fa3

Move benchmark images under static/img, escape YCSB for spell check, and replace internal cluster name with plain-language wording. Co-authored-by: Cursor <cursoragent@cursor.com>


		## 2. Documentation

		[User Documentation](https://github.com/apache/ozone/blob/HDDS-10685/hadoop-hdds/docs/content/feature/Short-Circuit-Read.md) of Short Circuit Read has been added.


		## 9. Possible incompatible changes/used feature flag

		Short-circuit read is gated by the non-rolling upgrade framework as HDDSLayoutFeature.SHORT_CIRCUIT_READS (layout version 11).


		End-to-end read performance is therefore dominated by local disk I/O, similar to a direct file read, rather than by gRPC data transfer.

		A benchmark was run against feature branch HDDS-10685, and get this result https://docs.google.com/document/d/1xmjCbK4rP287pPN0_iOonnQee6HgwSI3h5hVbtgfu3k/edit?tab=t.0.


		Short-circuit communication uses a UNIX domain socket (`ozone.domain.socket.path`) between the client and Datanode.

		It follows the same [Hadoop Socket Path Security](https://cwiki.apache.org/confluence/spaces/HADOOP2/pages/120730260/SocketPathSecurity) rules as HDFS short-circuit reads.

Uh oh!

Conversation

ChenSammi commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

What is the link to the Apache Jira?

How was this patch tested?

Uh oh!

Gargi-jais11 left a comment

Choose a reason for hiding this comment

Uh oh!

priyeshkaratha Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

priyeshkaratha left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ChenSammi commented Jul 24, 2026

Uh oh!

priyeshkaratha left a comment

Choose a reason for hiding this comment

Uh oh!

errose28 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ChenSammi Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

errose28 Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jojochuang Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

ChenSammi commented Jul 22, 2026 •

edited

Loading

priyeshkaratha Jul 23, 2026 •

edited

Loading

priyeshkaratha left a comment •

edited

Loading

ChenSammi Jul 28, 2026 •

edited

Loading

errose28 Jul 28, 2026 •

edited

Loading

jojochuang Jul 24, 2026 •

edited

Loading